1. Field of the Invention
The present invention relates to a neural network circuit constructed by connecting a multiplicity of neuron circuits formed by synapse calculation circuits, summation circuits, and nonlinear units, and a processing scheme using such a neural network circuit.
2. Description of the Background Art
A neural network circuit generally comprises a multiplicity of neuron circuits connected together, where each neuron circuit is a circuit with a plurality of inputs (x1, x2, . . . , xn) and one output (y) as shown in FIG. 1. In this neuron circuit of FIG. 1, a plurality of synapse weights (w1, w2, . . . , wn) are provided in correspondence to a plurality of inputs, and each synapse calculation circuit 101 carries out a calculation for obtaining a product of each input value and each synapse weight value, an absolute value of a difference between each input value and each synapse weight value, or a square of a difference between each input value and each synapse weight value, etc., while a summation circuit 102 sums calculation results obtained by all the synapse calculation units 101, and then a nonlinear unit 103 determines an output value according to a size of a summation result obtained by the summation circuit 102.
Here, the nonlinear unit 103 which finally determines the output value has one of transfer characteristics as shown in FIGS. 2A to 2D. Among them, the sigmoid function form of FIG. 2A has the most general applicability, but it is also possible to use the simplified forms such as a polygonal line form as shown in FIG. 2B or a step function form as shown in FIG. 2C for the purpose of simplifying the calculations, or a non-monotonic form as shown in FIG. 2D for the purpose of improving the functionality of the neural network circuit. In FIGS. 2A to 2D, symbols Th, Th1, Th2, Th3, and Th4 indicate saturation region boundary values in the respective figures which separate the saturation regions in which the output is constant at either Hi or Low level and transient regions in which the output changes according to the input.
The specific structure of the neural network circuit is determined by the connections of such neuron circuits. The most generally used structure is a triple layer structure shown, in FIG. 3, which comprises an input layer formed by a plurality of input terminals x1 to xn, a hidden layer formed by a plurality of neuron circuits 111, and an output layer formed by a plurality of neuron circuits 112. The signal entered from the input terminals x1 to xn are entered into all the neuron circuits 111 of the hidden layer in parallel and processed in parallel, while the outputs of all the neuron circuits 111 of the hidden layer are entered into all the neuron circuits 112 of the output layer in parallel and processed in parallel. Whenever the input signal is entered, the specific neuron circuits react to the entered input signal and realize the processing such as that of the recognition.
Conventionally, in a case of realizing a large scale neural network circuit having the above described function, it has been customary to construct the neural network circuit by combining a micro-processor 121 and RAMs 122 through a bus 123 as shown in FIG. 4. In such a configuration, the synapse calculations, the summation, and the nonlinear processing are carried out by the micro-processor 121, and the synapse weights are stored in the RAMs 122. During the calculations at the micro-processor 121, whenever the synapse weights become necessary, the accesses to the RAMs 122 are made to read out the necessary synapse weights.
In such a configuration, the scale of the realizable neural network circuit can be enlarged by increasing a number of RAMs 122, and the calculation algorithm can be changed easily by changing the program installed on the micro-processor 121.
However, in this conventional configuration, the processing speed is restricted by the access speed with respect to the synapse weights in the RAMs 122. In addition, the amount of calculations for the synapse calculations and the summation increases as the scale of the neural network circuit becomes larger, and this in turn restricts the processing speed for the following reason. Namely, in this conventional configuration, the synapse calculations with respect to all bits of the input value and all bits of the synapse weight for all synapse connections as well as the summation of all the synapse calculation results must be carried out, so that the amount of calculations can be quite enormous and the accesses to the synapse weights in the RAMs 122 associated with these calculations can also take considerable amount of time.
In order to cope with this problem concerning the amount of calculations required in each neuron circuit, the following two propositions have been made conventionally.
(1) A scheme disclosed in Japanese Patent Application Laid Open No. 4-112362 (1992) and U.S. Pat. No. 5,166,539 in which a difference or a distance between each input and each synapse weight, or a weight value due to each synapse weight is obtained as a weight calculation value of a single polarity by a weight calculation circuit, and all the weight calculation values are summed, and then the nonlinear processing is applied to this summation result to obtain an appropriate output, where an intermediate result of the summation calculation and the saturation region boundary values of the nonlinear unit are compared to judge whether or not to continue the calculations for the remaining inputs such that the summation calculations for the remaining inputs can be omitted whenever the intermediate result reached to the saturation region, while the intermediate result requiring the further summation calculations is selected by a selective control circuit, so as to reduce the circuit scale and the power consumption of the summation circuit. This scheme, however, is applicable only to a case in which the weight calculation result of the input and the synapse weight takes a positive value, so that its applicability is rather limited.
(2) A scheme disclosed in Y. Kondo, et al.: "A 1.2GFLOPS Neural Network Chip Exhibiting Fast Convergence", 1994 IEEE International Solid-State Circuit Conference, Digests of Technical Papers, pp. 218-219, February, 1994, in which the weight calculation using the synapse weight with a value less than a certain prescribed value is omitted in order to increase the processing speed. This scheme, however, is associated with a problem that the accurate summation calculation result may not be obtainable due to the accumulation of small errors due to the omitted weight calculations when the weight calculation values are summed.